Learning Dynamic Manipulation Skills under Unknown Dynamics with Guided Policy Search

Planning and trajectory optimization can readily be used for kinematic control of robotic manipulation. However, planning dynamic motor skills requires a detailed physical simulation, and some aspects of the task, such as contacts, are very difficult to simulate with enough accuracy for dynamic manipulation. Alternatively, manipulation skills can be learned from experience, allowing them to deftly exploit the dynamics of the real world. This is the approach taken in reinforcement learning [12, 14, 6], where a control policy is optimized using experience gathered directly with the robot. However, applying reinforcement learning to realistic robotics tasks typically requires a carefully engineered policy class with a modest number of parameters to make the learning task tractable [3]. Recently developed guided policy search methods can be used to learn general-purpose controllers represented by neural networks, without task-specific engineering, by using trajectory optimization to discover successful task executions [7, 8, 9]. These methods previously required a simulator in order to perform trajectory optimization, making them difficult to apply to robotic motor skill learning. We present a trajectory optimization algorithm suitable for use with guided policy search that does not require a known dynamics model or simulator. Our experimental evaluation shows that this approach can optimize manipulation trajectories that are extremely challenging for previous reinforcement learning methods. We also show that, combined with guided policy search, our method can learn complex policies for a simulated peg insertion task in a partially observed environment.